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Since the early 1980s, personal computers have become 
increasingly powerful, able to store large amounts of 
information, create complex text, and multimedia docu- 
ments including now color animation, 3D effects, and sound. 
In addition, these devices are able to communicate over 
telephone lines or local area networks with other computers 
directly or through the Internet. These computers are able to 
draw on large databases stored on large capacity hard drives 
in the personal computers. PCs can also tap into remote 
databases through their networking and communications 
capabilities. 

Although the human interface to these computers has 
evolved to a certain extent from the early 1980s, in many 
ways the major element of this interface, the keyboard, is 
still very similar to that of a manual typewriter whose origins 
date to the late part of the 19th Century. For most computers, 
even in the mid-990s, the 100-key keyboard with alpha/ 
numeric and function keys still forms the basic input means 
for accessing and creating information on personal and other 
computers. Ironically, the keyboard that is in common use 
has its basic layout designed to slow typists down. This 
design dates from the days of mechanical typewriters whose 
keys jammed when typists became too proficient. Although 
many people using computers have learned to type very 
rapidly, for many who do not learn to type well or who do 
not know how to type, the keyboard interface to the com- 
puter represents a barrier to its use. In addition, many people 
who do learn to type well can develop a repetitive stress 45 
disorder, an inflammation of the wrists which can result in 
the complete inability to type and therefore loss of produc- 
tivity on the computer. 

In the late 1980s a pointing device, called a mouse, was 
developed for computer input which allows the user to move 50 
a curser or indicator within the computer output display 
screen. By pointing and clicking a mouse, certain words or 
areas on the screen may be chosen by the user. In this way, 
navigation of the display screen and command of computer 
operations may be controlled by pointing to various items or 55 
words or icons on the screen. The pointing device may be a 
mouse, which indirectly points to items on the screen, or a 
pen-type device applied directly to the screen or even a 
finger with a special touch screen. 

Other operations are possible using these devices such as 60 
highlighting a word in order to provide an additional com- 
mand by means of other switches on the pointing device to 
delete the word or change its appearance. The development 
of the graphic user interfaces (GUI), have greatly enhanced 
the use of pointing devices for the human interface to the 65 
computer. Although these pointing devices may substitute 
for a series of keystrokes for moving a pointer around on the 
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significant computing power lor their operation. Early 
speech recognition devices could be trained by an individual 
to respond to a small number of command words effectively 
substituting for command keys on the keyboard or a Limited 
number of mouse clicks in a Windows interface. As com- 
puters have become more powerful in their computing speed 
and memory capacity, automatic speech recognition systems 
for computer input have become more capable. It is possible 
on personal computers to use voice input commands to 
activate any Windows command that appears in the menu 
structure using discrete or continuous speech recognition 
without requiring navigation through several layers of 
menus. Speech recognition systems are an especially pow- 
erful substitute for the keyboard for the input of individual 
words of text to create documents or for discrete commands. 
Such systems, however, are not a good substitute for the ease 
and speed of display screen navigation or other drawing 
operations (for example circling a block of text and moving 
it by dragging it to a new place on the screen), which can 
easily be provided by a mouse or other pointing device. 
Moreover, such speech recognition systems have difficulty 
determining whether the received speech is a command or 
text. 

Although the promise of automatic speech recognition 
systems for text creation using computers is great because 
they are rapid and easy to use, these systems suffer from 
some significant limitations which have impeded their gen- 
eral use in computers. The accuracy of speech recognition 
systems, even those well trained to the voice of a single user, 
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The physical size of computers has limited their utility in 
certain applications. Like many electronic devices, comput- 
ers have grown dramatically smaller as they have evolved. 
In recent years, laptop and even palmtop computers the size 
of small books have become popular. A computer the size of 
a book, which may be carried anywhere or a small pocket- 
sized device, has no room for a keyboard large enough to 
accommodate hands of most adults. In addition, if a com- 
puter is to be used in the field as a palmtop device or even 
in an airplane seat, the use of a mouse-type pointing device 
that requires an external pad is impractical. A pointing 
device such as a pen for use on even a small computer 
display surface is extremely useful. 

A number of devices without keyboards have been pro- 
posed that use pens and have handwriting recognition as 
input and/or receive mouse-type input. Those introduced 
have had limited ability to recognize even fairly clear 
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handwriting. Although handwriting recognition by pen input FIG .,1 is a block diagram of computer sy&em^UK) for 
devices has significantly improved in the last few years, like processing handwritten or hand-drawn -input' and speech 
speech recognition, it still remains a challenging technical input comprising mode controller 102, mode processing 
problem. For example, pen input in currently available logic m i[ltcr f acc controller 106, speech interface 108, pen- 
systems is tinng and impractical when entering large s mterf no and application progra^Tl6 ^MicTOphorle 
amounts of tex Developing even smaller personal comput- ^ ^ £ ^ -ElecWic ~n^ll4 
mg devices with the complete text input and computing . .-2r <Va . ^ 

° . * r » i . r . connects to pen interface 110. 

capability of larger sized personal computers remains a ^ 

major goal and interest of the computing public and the Interface controller 106 controls speech interface 108 and 

computing industry. pen interface 110, providing pen or .speech input to mode 

There is, therefore, a need for a computer system that 10 controller 102. Speech interface ltW preferably includes 

departs from conventional methods and achieves increased computer hardware and software for encoding an electronic 

performance by integrating speech recognition and hand- signal generated by microphone 112 into a digital stream for 

written and hand-drawn (e.g., pen or gesture input) recog- processing by mode processing logic 104. Similarly, pen 

nition to overcome the disadvantages of either mode of interface 110 preferably includes computer hardware and 

recognition used alone or in an un integrated combination. * 35 software for processing handwritten or hand -drawn input 

SUMMARY OF THE INVENTION created ^ electronic P en 114 
Methods and apparatus consistent with this invention . Modecontroller 102 activates modes in moo^ processing 
process handwritten or handKlrawn input and speech input. ^'ding to mput received from mterface con. 
Method steps include recognizing received handwritten or M ^ er A 106 to cr ? ate an oP^ahng state for computer system 
hand-drawn [input, recognizing received speech input, and 20 100. An operating state governs how mput received from 
creating or modifying an electronic document according to interface controller 106 is processed and passed to applica- 
ble speech or handwritten or hand-drawn input. *on programs 116. Application programs 116 include, for 
An apparatus includes structure for recognizing handwrit- example computer programs for creating, editing, and view- 
ten or hand-drawn input, structure for recognizing speech electronic documents, such as word processing, graphic 
input, and structure for activating modes for processing the design, spreadsheet, electronic mail, and web browsing 
handwritten or hand-drawn input and the speech input programs. 

responsive to handwritten or hand-drawn input or the speech An operating state is defined by one or more active modes 

input. in mode processing logic 104. Mode processing logic 104 

Both the foregoing general description and the following preferably supports five modes. The processing logic in 

detailed description are exemplary and explanatory and are 30 mode Ml recognizes speech input to microphone 112. In 

intended to provide further explanation of the invention as mo d e Ml, computer system 100 translates the speech input 

claimed. into a machine encoded (e.g., ASCII) text stream. For 

DESCRIPTION OF THE FIGURES example, in mode Ml, computer system 100 translates 

„ , j a .1 continuous speech spoken by an operator into microphone 

The accompanying drawings provide a further under- 35 . *\. * . J , [\ c r c 

* j ■ c ,u • .* tv •n.„.,„,„ a ^ n A; man f* 112 into machine encoded text data forming part of an 

standing of the invention. They illustrate embodiments ot & r 

the invention and, together with the description, explain the electronic document. 

principles of the invention. In mode M2, the processing logic recognizes pen mput, 

FIG. 1 is a block diagram of a computer system for i.e., it translates handwritten or hand-drawn input received 

processing handwritten or hand-drawn input and speech d0 from electronic pen U4 into a machine encoded text stream, 

input. For example, In mode M2, computer system 100 translates 

FIG. 2 is a flow diagram of a method for processing hand-printed or cursive writing written by an operator using 

speech and handwritten input; electronic pen 114 into machine encoded text data forming 

FIG. 3 is a flow diagram of a second method for process- part of an electronic document, 

ing speech and handwritten input; 45 In mode M3, the processing logic recognizes hand-drawn 

FIG. 4 is a flow diagram of a method for processing graphic images created with electronic pen 114. For 

hand-drawn input and speech input; example, in mode M3, computer system 100 recognizes an 

FIG. 5 is a flow diagram of a method for processing attempt by the operator to draw a rough outline for a circle, 

handwritten input and speech input; triangle, or square. The processing logic in mode may M3 

FIG. 6 is a flow diagram of a method for processing 50 create a corresponding "clean" (e.g., smooths curves, 

handwritten input and speech input; straightens lines, and corrects any geometric distortions) 

FIG. 7 is a flow diagram of a method for editing an electronic representation of the drawing by issuing a snap- 
electronic document; to-grid command. 

FIG. 8 is a flow diagram of a method for processing \ n mo de M4, the processing logic treats input received 

hand-drawn input and speech input; 55 f rom electronic pen 114 as a command or an indication of a 

FIG. 9 is a block diagram of a computer system for position in an electronic display. In mode M5, the processing 

processing gesture, handwritten, or hand-drawn input and logic recognizes speech input received from microphone 112 

speech input; and as a command: The commands recognized in modes M4 and 

FIG. 10 is a block diagram of a computer system with a ^5 include, for example, commands or menus that are 

mode switch for processing gesture, handwritten, or hand- 6Q acce ssible from tool bars and pull-down menus in many 

drawn input and speech input. graphics and text application programs. 

DETAILED DESCRIPTION OF THE A "combination mode" consists of two or more active 

INVENTION modes. One such combination mode is an operating state 

Reference will now be made in detail to preferred where modes Ml and M2 are active. FIG. 2 is a flow- 
embodiments illustrated in the accompanying drawings. The 65 diagram of a method for processing speech and handwritten 
same numbers in different figures refer to like or similar input. According to the method shown in FIG. 2, the 
elements. computer system receives speech input by an operator (step 
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202). Next the computer system generates a preliminary method for processing handwritten input and speech input, 

recognition of the speech input using mode Ml (step 204). In this operating state, mode Ml processing logic recognizes 

For example, the computer's preliminary classification speech spoken into microphone 114 (step 601) and displays 

would identify several text strings, letters, characters, or a. corresponding text string on a computer display 1 (not 

words corresponding to the closest matches to the recog- 5 shown) connected to computer system 100 (step 602). The 

nized speech input. The operator then enters handwriting operator using electronic pen 114 makes .gestures, for 

input providing the computer system with additional data to example, directly on the display .or on an. electronic tablet' 

improve recognition of the input speech (step 206). Using that are recognized by M4 pro cessing tdgic as commands for 

mode M2, computer system 100 generates an improved modifying the text (step 604). (Any voice commands would 

recognition of the input speech from additional data in mode 1Q be handled by M5 processing logic when'the'jVW mode "is 

M2 to select from among recognition candidates identified activated.) Computer system 100 then applies the command 

by mode Ml at step 204. to foe tex t in accordance with an established mapping" of 

FIG. 3 is a flow diagram of a method for processing gestures to commands (step 606). Examples *of these 
speech and handwritten input where speech input is used to gesture-to-command mappings include circling a word to 
enhance the interpretation of handwriting or hand-drawn indicate highlighting, which activates combination mode 
input and speech input (multimodal input). According to the M4-M5, thereby switching the system from state M1-M5 to 
method shown in FIG. 3, computer system 100 receives sUt6 M4 ^j 5 A spokcn command such as "bold" accom- 
handwriting input (step 302) and speech input from an tfae ^ A keQ command( such ^ "QIC- 
operator (step 304 . Next system 100 generates a prckmi- letes me proccss (step 607)> uAnother example of 
nary recognition of the multimodal input (step 306) Using stu H re „to-command mapping is scratching through a word 
mode Ml and M2, the computer system generates an 20 8 the pen to delete itTaaivalmg state 
enhanced interpretation of the input using the additional data w - * n a* u *7 • 1 ~f t*u 
provided by both the handwritten or hand-drawn input and M1-M2 that allows a new word to be written in place of the 
input speech (step 308). scratched out word. 

In a second combination mode, the operating state has In a fifth combination mode, the operating state has both 

both modes Ml and M3 active. FIG. 4 is a flow diagram of 25 modes M4 and MS actlve * FIG. 7 is a flow diagram of a 

a method for processing hand-drawn input and speech input. method for editing an electronic document m this operating 

Computer system 100 receives a geometric shape or figure state. Computer system 100 receives speech input from an 

drawn by the operator using electronic pen 114 (step 402). operator specifying an editing command (step 702). Using 

System 100 generates a graphic corresponding to the hand- MS processing logic, computer system 100 recognizes a 

drawn geometric shape using processing mode M3 (step 30 command in the speech input (step 704). Computer system 

404). The location of the electronic pen defines an insert 100 also receives gesture input from the operator indicating 

position in an electronic document for text recognized from a command that can also include the spatial coordinates of 

the speech input (step 405). The computer system then the portion of the electronic document to be edited (step 

receives speech input corresponding to text to be placed in 706). Computer system 100 M4 processing logic recognizes 

the electronic document (step 406). Using processing mode 35 a command in the gesture input (step 708). Computer system 

Ml, computer system 100 generates a text string from the 100 then modifies the electronic document according to 

speech input (step 408). System 100 places the text where either or both the command from the speech input and the 

indicated by the pen (step 410). The foregoing method is command from the gesture input (step 710). 

useful when creating annotated charts, for example, corpo- Using computer system 100 in this operating state is 

rate organization charts. Using this method, an operator uses 40 advantageous when an operator desires to modify an elec- 

an electronic pen to sketch rough outlines of boxes corrc- tronic document, such as a drawing created in a computer 

sponding to the organization structure, the computer cleans- graphics application program or a document created in a 

up the boxes, and the user dictates annotations that are wor£ j processing application program. For example, to 

automatically inserted where indicated by the pen in the modify a drawing in a graphics editing context using elec- 

figure. 45 tronic pen 114, the operator would circle a portion of the 

In a third combination mode, the operating state has both drawing displayed on a computer screen. The operator 

modes M2 and M5 active. FIG, 5 is a flow diagram of a would then say "change color from red to blue "Application 

method for processing handwritten input and speech input. program 116 then responds to the spoken and gesture 

Using a computer input device, for example, electronic pen commands and changes the color of the circled region in the 

114, an operator writes text that computer system 100, using 50 drawing from red to blue. 

processing logic in mode M2, recognizes and generates an i n a s i x t n combination mode, the operating state has both 

electronic document (step 502). To format or modify the modes M3 and M5 active. FIG, 8 is a flow diagram of a 

document, the operator speaks commands into microphone method for processing hand-drawn input and speech input. 

112 that are recognized by computer system 100 using Using processing logic in mode M3, Computer system 100 

processing logic in mode M5 (step 504). Computer system 55 recognizes hand-drawn input from electronic pen 114 cor- 

100 modifies the electronic document as instructed by the responding to, for example, graphic images (circles, 

command (step 506). triangles, squares, line drawings, etc.) (step 802). Mode 

Using computer system 100 in an operating state with processing logic in mode M5 recognizes a command in the 

modes M2 and M5 active, an operator can create a memo by speech input for formatting the graphic image (step 804). 

printing or cursive writing and have the words recognized 60 Application program 116 formats the graphic image accord - 

and displayed on a computer screen. To format the memo as ing to the command (step 806). This operating state allows 

it is being created, the operator can utter commands into the an operator to create drawings quickly without navigating 

microphone such as "bold," ''underline/' "paragraph/* etc. through complex pull-down menus. For example, an opera- 

The electronic document that is displayed on the screen will tor sketches the outline of a square with electronic pen 114 

change in response to these commands. 65 and says "fill color black, cross-hatch, dashed outline." 

In a fourth combination mode, the operating state has both Computer system 100 then creates a square (straightens the 

modes Ml and M4 active. FIG. 6 is a flow diagram of a lines drawn by the operator, makes them of equal length, and 
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connects them at right angles) and formats it according to the 
spoken commands. 

The foregoing five modes are not exhaustive of all pos- 
sible modes. Moreover, the six combinations of modes is 
only illustrative of a limited number of the many possible 
combinations of the modes for an operating state. Therefore, 
this invention is not limited to the particular modes and 
combinations disclosed, but includes all modes and combi- 
nations falling within the scope of the claims. 

Id an embodiment, computer system 100 mode controller 
102 switches between modes and combination modes 
thereby switching operating ^states^Switcbing operating 
states changes the manner in which structure for recognizing 
gesture input recognizes gestures, handwritten, and hand- 
drawn input, and the structure for recognizing speech input 
recognizes speech. For example, as described in greater 
detail above, switching among modes and combinations of 
modes will control whether the recognition process will treat 
gesture input as a command, text, or drawing, and whether 
speech input is recognized as a command or text-Switching 
among operating states can be controlled by, for example, 
application program 116, gestures received by pen interface 
110, speech received by speech interface 108, or an elec- 
tronic or mechanical switch. A switch or button can be 
connected to computer system 100, microphone 112, elec- 
tronic pen 114, or any other peripheral device associated 
with computer system 100 (e.g., a digitizing tablet con- 
nected to computer system to control switching among 
operating states. Other types of controls for switching oper- 
ating states include rotating wheels, numeric keypads, and 
chorded keypads for one-handed letter input. FIG. 10 is a 
block diagram of a computer system with switch 1002 for 
processing handwritten and hand-drawn input and speech 
input. 

Methods and apparatus for processing speech and hand- 
written and hand -drawn input are suitable for several appli- 
cation environments including, but not limited to, informa- 
tion kiosks, television/video cassette .recorder remote 
control, a low profile computer with a form factor similar to 4Q 
a pad of paper (e.g., slate or pen tablet computer), a palm 
computer, a telephone, an electronic whiteboard, or a hand- 
held personal computer. 

Computer system 100 can receive several types of devices 
for providing gesture interaction with mode processing logic 45 
104. FIG. 9 is a block diagram of computer system 100 for 
processing gesture, handwritten, and hand-drawn input and 
speech input comprising several of the elements described in 
greater detail above in FIG. 1, and wireless gesture interface 
902. Pointing device 904 is an electronic glove that tracks 
and transmits movement of the operator's hand. 
Alternatively, pointing device 904 could be a three- 
dimensional wireless mouse or wand. In yet another 
implementation, structure for providing gesture interface 
902 includes a video tracking device, for example, the 
interface made by Reality Fusion that tracks the movement 
of the operator's hand as it makes gestures in the air. 

A computer display can be connected to, or integrated 
within, computer system 100. To provide the operator with 
feedback regarding the operating state, the appearance of the 
display changes in accordance with a change in operating 
state. For example, in a combination mode with Ml and M4 
active, the display would have a white background. Chang- 
ing the operating state to a combination mode with M2 and 
M5 active would display ruled or grid lines on the back- 
ground of the display. Similarly, a background color change 
would indicate yet another operating state. 
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While some embodiments have been described, various 
changes and modifications may be made, and equivalents 
may be substituted. In addition, many modifications may be 
made to adapt a particular element, technique or implemen- 
tation to the teachings of the present invention without 
departing from the central scope of the invention. For 
example, disclosed elements may be implemented in 
hardware, computer program code, or a combination of both 
hardware and computer program code. Moreover, elements 
depicted and described separately may be combined and 
implemented in a single element or distributed across a 
computer network. Therefore, this invention is not limited to 
the particular embodiments and methods disclosed, but 
includes all embodiments falling within the scope of the 
appended claims. 

We claim: 

1. A method for processing handwritten input or hand- 
drawn input and speech input comprising the steps, per- 
formed by a computer, of: 

selecting a first operating mode and a second operating 
mode before processing handwritten input, hand-drawn 
input, or speech input; 

recognizing the speech input as either a command or text 
for an electronic document in accordance with said first 
operating mode; 

recognizing the handwritten or hand-drawn input as a 
command, text for an electronic document, or a graphic 
for an electronic document in accordance with said 
second operating mode; and 

switching said first operating mode or said second oper- 
ating mode to change how the computer recognizes 
said speech input or said handwritten or hand-drawn 
input. 

2. The method of claim 1, wherein the step of switching 
said first operating mode or said second operating mode 
includes the substep of switching either said first mode or 
said second mode in response to a gesture. 

3. The method of claim 1, wherein the step of switching 
said first operating mode or said second operating mode 
includes the substep of switching either said first mode or 
said second mode in response to speech input. 

4. The method of claim 1, wherein the step of switching 
said first operating mode or said second operating mode 
includes the substep of switching either said first mode or 
said second mode in response to a mechanical switch 
activated by an operator. 

5. A method for processing handwritten input or hand- 
drawn input and speech input comprising the steps, per- 
formed by a computer, of: 

selecting a first combination mode and a second combi- 
nation mode before processing handwritten hand- 
drawn, or speech input; 

processing said handwritten input or said speech input in 
said first combination mode wherein said speech input 
is recognized as text input and said handwritten input is 
recognized as a command; 

processing said handwritten input or said speech input in 
said second combination mode wherein said speech 
input is recognized as text and said handwritten input is 
recognized as text; and 

switching processing modes between said first combina- 
tion mode and said second combination mode. 

6. A method for processing handwritten input or hand- 
drawn input and speech input comprising the steps, per- 
formed by a computer, of: 

selecting a first combination mode and a second combi- 
nation mode before processing handwritten, hand- 
drawn, or speech input; 
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processing said handwritten input or said speech input in 
said first combination mode wherein said speech input 
is recognized as text input and said handwritten input is 
recognized as a command; 

processing said handwritten input or said speech input in 
said second combination mode wherein said speech 
input is recognized as a command and said handwritten 
input is recognized as a command; and 

switching processing modes between said first combina- 
tion mode and said second combination mode. 

7. A method for processing handwritten input or hand- 
drawn input and speech input comprising the steps, per- 
formed by a computer, of: 

selecting a first combination mode and a second combi- 
nation mode before processing handwritten, hand- 
drawn, or speech input; 

processing said handwritten input or said speech input in 
said first combination mode wherein said speech input 
is recognized as text input and said handwritten input is 
recognized as a command; 

processing said handwritten input or said speech input in 
said second combination mode wherein said speech 
input is recognized as a command and said handwritten 
input is recognized as text; and 

switching processing modes between said first combina- 
tion mode and said second combination mode. 

8. A method for processing handwritten input or hand- 
drawn input and speech input comprising the steps, per- 
formed by a computer, of: 

selecting a first combination mode and a second combi- 
nation mode before processing handwritten, hand- 
drawn, or speech input; 

processing said handwritten input or said speech input in 
said first combination mode wherein said speech input 
is recognized as text and said handwritten input is 
recognized as text; 

processing said handwritten input or said speech input in 
said second combination mode wherein said speech 
input is recognized as a command and said handwritten 
input is recognized as a command; and 

switching processing modes between said first combina- 
tion mode and said second combination mode, 

9. A method for processing handwritten input or hand- 
drawn input and speech input comprising the steps, per- 
formed by a computer, of: 

selecting a first combination mode and a second combi- 
nation mode before processing handwritten, hand- 
drawn, or speech input; 

processing said handwritten input or said speech input in 
said first combination mode wherein said speech input 
is recognized as text and said handwritten input is 
recognized as text; 

processing said handwritten input or said speech input in 
said second combination mode wherein said speech 
input is recognized as a command and said handwritten 
input is recognized as text; and 

switching processing modes between said first combina- 
tion mode and said second combination mode. 

10. A method for processing handwritten input or hand- 
drawn input and speech input comprising the steps, per- 
formed by a computer, of: 

selecting a first combination mode and a second combi- 
nation mode before processing handwritten, hand- 
drawn, or speech input; 
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processing said handwritten input or said speech input in 
said first combination mode wherein said speech input 
is recognized as a command and said handwritten input 
is recognized as text; 

processing said handwritten input or said speech input in 
said second combination mode wherein said speech 
input is recognized as a command and said handwritten 
input is recognized as a command; and 

switching processing modes between said first combina- 
tion mode and said second combination mode. 

11. A method for processing handwritten input or hand- 
drawn input and speech input comprising the steps, per- 
formed by a computer, of: 

selecting a first combination mode and a second combi- 
nation mode before processing handwritten, hand- 
drawn, or speech input; 

processing said handwritten input or said speech input in 
said first combination mode wherein said speech input 
is recognized as text input and said handwritten input is 
recognized as a command; 

processing said handwritten input or said speech input in 
said second combination mode wherein said speech 
input is recognized as text input and said handwritten 
input is recognized as a graphic; and 

switching processing modes between said first combina- 
tion mode and said second combination mode. 

12. A method for processing handwritten input or hand- 
drawn input and speech input comprising the steps, per- 
formed by a computer, of: 

selecting a first combination mode and a second combi- 
nation mode before processing handwritten, hand- 
drawn, or speech input; 

processing said handwritten input or said speech input in 
said first combination mode wherein said speech input 
is recognized as text input and said handwritten input is 
recognized as a command; 

processing said handwritten input or said speech input in 
said second combination mode wherein said speech 
input is recognized as a command and said handwritten 
input is recognized as a graphic; and 

switching processing modes between said first combina- 
tion mode and said second combination mode. 

13. A method for processing handwritten input or hand- 
drawn input and speech input comprising the steps, per- 
formed by a computer, of: 

selecting a first combination mode and a second combi- 
nation mode before processing handwritten, hand- 
drawn, or speech input; 

processing said handwritten input or said speech input in 
said first combination mode wherein said speech input 
is recognized as text and said handwritten input is 
recognized as text; 

processing said handwritten input or said speech input in 
said second combination mode wherein said speech 
input is recognized as text input and said handwritten 
input is recognized as a graphic; and 

switching processing modes between said first combina- 
tion mode and said second combination mode. 

14. A method for processing handwritten input or hand- 
drawn input and speech input comprising the steps, per- 
formed by a computer, of: 

selecting a first combination mode and a second combi- 
nation mode before processing handwritten, hand- 
drawn, or speech input; 



10/28/2003, EAST Version: 1.4.1 



US 6,438,523 Bl 



11 



12 



processing said handwritten input or said speech input in 
said first combination mode wherein said speech input 
is recognized as text and said handwritten input is 
recognized &s text; 

processing said handwritten input or said speech input in 
said second combination mode wherein said speech 
input is recognized as a command and said handwritten 
input is recognized as a graphic; and 

switching processing modes between said first combina- 
tion mode and said second combination mode. 

15. A method for processing handwritten input or hand- 
drawn input and speech input comprising the steps, per- 
formed by a computer, of: 

selecting a first combination mode and a second combi- 
nation mode before processing handwritten, hand- 
drawn, or speech input; 

processing said handwritten input or said speech input in 
said first combination mode wherein said speech input 
is recognized as text input and said handwritten input is 
recognized as a graphic; 

processing said handwritten input or said speech input in 
said second combination mode wherein said speech 
input is recognized as a command and said handwritten 
input is recognized as a command; and 

switching processing modes between said first combina- 
tion mode and said second combination mode. 

16. A method for processing handwritten input or hand- 
drawn input and speech input comprising the steps, per- 
formed by a computer, of: 

selecting a first combination mode and a second combi- 
nation mode before processing handwritten, hand- 
drawn, or speech input; 

processing said handwritten input or said speech input in 
said first combination mode wherein said speech input 
is recognized as text input and said handwritten input is 
recognized as a graphic; 

processing said handwritten input or said speech input in 
said second combination mode wherein said speech 
input is recognized as a command and said handwritten 
input is recognized as text; and 

switching processing modes between said first combina- 
tion mode and said second combination mode. 

17. A method for processing handwritten input or hand- 
drawn input and speech input comprising the steps, per- 
formed by a computer, of: 

selecting a first combination mode and a second combi- 
nation mode before processing handwritten, hand- 
drawn, or speech input; 

processing said handwritten input or said speech input in 
said first combination mode wherein said speech input 
is recognized as text input and said handwritten input is 
recognized as a graphic; 

processing said handwritten input or said speech input in 
said second combination mode wherein said speech 
input is recognized as a command and said handwritten 
input is recognized as a graphic; and 

switching processing modes between said first combina- 
tion mode and said second combination mode. 

18. A method for processing handwritten input or hand- 
drawn input and speech input comprising the steps, per- 
formed by a computer, of: 

selecting a first combination mode and a second combi- 
nation mode before processing handwritten, hand- 
drawn, or speech input; 
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processing said handwritten input or said speech input in 
said first combination mode wherein said speech input 
is recognized as a command and said handwritten input 
is recognized as a command; 

processing said handwritten input or said speech input in 
said second combination mode wherein said speech 
input is recognized as a command and said handwritten 
input is recognized as a graphic; and 

switching processing modes between said first combina- 
tion mode and said second combination mode. 

19. A method for processing handwritten input or hand- 
drawn input and speech input comprising the steps, per- 
formed by a computer, of: 

selecting a first combination mode and a second combi- 
nation mode before processing handwritten, hand- 
drawn, or speech input; 

processing said handwritten input or said speech input in 
said first combination mode wherein said speech input 
is recognized as a command and said handwritten input 
is recognized as text input; 

processing said handwritten input or said speech input in 
said second combination mode wherein said speech 
input is recognized as a command and said handwritten 
input is recognized as a graphic; and 

switching processing modes between said first combina- 
tion mode and said second combination mode. 

20. A pen tablet computer for processing handwritten 
input or hand-drawn input and speech input comprising: 

selecting a first operating mode and a second operating 
mode before processing handwritten, hand-drawn, or 
speech input; 

means for recognizing the speech input as either a com- 
mand or text for an electronic document in accordance 
with said first operating mode; 

means for recognizing the handwritten or hand-drawn 
input as a command, text for an electronic document, or 
a graphic for an electronic document in accordance 
with said second operating mode; and 

means for switching said first operating mode or said 
second operating mode to change how the computer 
recognizes said speech input or said handwritten or 
hand -drawn input. 

21. The pen tablet computer of claim 20 further compris- 
ing a pen-integrated microphone. 

22. A palm computer for processing handwritten input or 
hand-drawn input and speech input comprising: 

selecting a first operating mode and a second operating 
mode before processing handwritten, hand-drawn, or 
speech input; 

means for recognizing the speech input as either a com- 
mand or text for an electronic document in accordance 
with said first operating mode; 

means for recognizing the handwritten or hand-drawn 
input as a command, text for an electronic document, or 
a graphic for an electronic document in accordance 
with said second operating mode; and 

means for switching said first operating mode or said 
second operating mode to change how the computer 
recognizes said speech input or said handwritten or 
hand-drawn input. 

23. The palm computer of claim 22 further comprising a 
pen-integrated microphone. 

24. A kiosk computer system for processing handwritten 
input or hand-drawn input and speech input comprising: 

selecting a first operating mode and a second operating 
mode before processing handwritten, hand-drawn, or 
speech input; 
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means for recognizing the speech input as either a com- 
mand or text for an electronic document in accordance 
with said first operating mode; 

means for recognizing the handwritten or hand-drawn 
input as a command, text for an electronic document, or 
a graphic for an electronic document in accordance 
with said second operating mode; and 

means for switching said first operating mode or said 
second operating mode to change how the computer 
recognizes said speech input or said handwritten or 
hand-drawn input. 

25. The kiosk computer system of claim 24, wherein said 
means for recognizing the handwritten or hand-drawn input 
includes either a touch screen, a digitizing tablet, or a video 
gesture recognizer. 

26. A computer for processing handwritten input or hand- 
drawn input and speech input comprising: 

selecting a first operating mode and a second operating 
mode before processing handwritten, hand-drawn, or 
speech input; 

means for recognizing the speech input as either a com- 
mand or text for an electronic document in accordance 
with said first operating mode; 

means for recognizing the handwritten or hand-drawn 
input as a command, text for an electronic document, or 
a graphic for an electronic document in accordance 
with said second operating mode; and 

means for switching said first operating mode or said 
second operating mode to change how the computer 
recognizes said speech input or said handwritten or 
hand-drawn input. 

27. The computer of claim 26, wherein said means for 
recognizing the handwritten or hand -drawn input includes 
either a touch screen, a digitizing tablet, a three-dimensional 
wireless mouse, or a video gesture recognizer. 

28. The computer of claim 26 further comprising 

a display with a background responsive to an operating 
state of either said means for recognizing the speech 
input or said means for recognizing the handwritten or 
hand-drawn input. 

29. The display of claim 28 further comprising 
means for changing screen tint. 

30. The display of claim 28 further comprising 
means for displaying lines. 

31. A television or video cassette recorder remote control 
for processing handwritten input or hand-drawn input and 
speech input comprising: 

selecting a first operating mode and a second operating 
mode before processing handwritten, hand-drawn, or 
speech input; 

means for recognizing the speech input as either a com- 
mand or text for an electronic document in accordance 
with said first operating mode; 

means for recognizing the handwritten or hand-drawn 
input as a command, text for an electronic document, or 
a graphic for an electronic document in accordance 
with said second operating mode; and 

means for switching said first operating mode or said 
second operating mode to change how the computer 
recognizes said speech input or said handwritten or 
hand-drawn input. 

32. The television or video cassette recorder remote 
control of claim 31, wherein said means for recognizing the 
handwritten or hand-drawn input includes either a three- 
dimensional wireless mouse or a video gesture recognizer. 
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33. An electronic whiteboard for processing handwritten 
input or hand-drawn input and speech input comprising: 

selecting a first operating mode and a second operating 
mode before processing handwritten, hand-drawn, or 
speech input; 

means for recognizing the speech input as either a com- 
mand or text for an electronic document in accordance 
with said first operating mode; 

means for recognizing the handwritten or hand-drawn 
input as a command, text for an electronic document, or 
a graphic for an electronic document in accordance 
with said second operating mode; and 

means for switching said first operating mode or said 
second operating mode to change how the computer 
recognizes said speech input or said handwritten or 
hand-drawn input. 

34. The electronic whiteboard of claim 33, wherein said 
means for recognizing the handwritten or hand-drawn input 
includes either a three-dimensional wireless mouse, a pen, a 
touch screen, or a video gesture recognizer. 

35. A computer for processing handwritten input or hand- 
drawn input and speech input comprising: 

means for recognizing handwritten or hand-drawn input; 

means for recognizing speech input; and 

a mechanical switch for switching among operating 

modes, wherein said operating modes include 

a first mode wherein said means for recognizing speech 
input recognizes speech as text, 

a second mode wherein said means for recognizing 
speech input recognizes speech as a command, 

a third mode wherein said means for recognizing 
handwritten or hand -drawn input recognizes hand- 
written input as text, 

a fourth mode wherein said means for recognizing 
handwritten or hand -drawn input recognizes hand- 
drawn input as a graphic, and 

a fifth mode wherein said means for recognizing hand- 
written or hand-drawn input recognizes handwritten 
or hand-drawn input as a command. 

36. A pen tablet computer for processing handwritten 
input or hand-drawn input and speech input comprising: 

means for recognizing handwritten or hand-drawn input; 

means for recognizing speech input; and 

a mechanical switch for switching among operating 

modes, wherein said operating modes include 

a first mode wherein said means for recognizing speech 
input recognizes speech as text, 

a second mode wherein said means for recognizing 
speech input recognizes speech as a command, 

a third mode wherein said means for recognizing 
handwritten or hand-drawn input recognizes hand- 
written input as text, 

a fourth mode wherein said means for recognizing 
handwritten or hand-drawn input recognizes hand- 
drawn input as a graphic, and 

a fifth mode wherein said means for recognizing hand- 
written or hand -drawn input recognizes handwritten 
or hand-drawn input as a command. 

37. A palm computer for processing handwritten input or 
hand-drawn input and speech input comprising: 

means for recognizing handwritten or hand-drawn input; 

means for recognizing speech input; and 

a mechanical switch for switching among operating 
modes, wherein said operating modes include 
a first mode wherein said means for recognizing speech 
input recognizes speech as text, 
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a second mode wherein said means for recognizing 
speech input recognizes speech as a command, 

a third mode wherein said means for recognizing 
handwritten or hand-drawn input recognizes hand- 
written input as text, 5 

a fourth mode wherein said means for recognizing 
handwritten or handwritten input recognizes hand- 
drawn input as a graphic, and 

a fifth mode wherein said means for recognizing hand- 
written or handwritten input recognizes handwritten 10 
or hand-drawn input as a command. 

38. A telephone for processing handwritten input or 
hand-drawn input and speech input comprising: 

means for recognizing handwritten or hand-drawn input; 
means for recognizing speech input; and 15 
a mechanical switch for switching among operating 
modes, wherein said operating modes include 
a first mode wherein said means for recognizing speech 

input recognizes speech as text, 
a second mode wherein said means for recognizing 

speech input recognizes speech as a command, 
a third mode wherein said means for recognizing 
handwritten or hand-drawn input recognizes hand- 
written input as text, 
a fourth mode wherein said means for recognizing 
handwritten or hand-drawn input recognizes hand- 
drawn input as a graphic, and 
a fifth mode wherein said means for recognizing hand- 
written or hand -drawn input recognizes handwritten 
or hand-drawn input as a command. 

39. A television or video cassette recorder remote control 
for processing handwritten input or hand-drawn input and 
speech input comprising: 

means for recognizing handwritten or hand-drawn input; 35 
means for recognizing speech input; and 
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a mechanical switch for switching among operating 

modes, wherein said operating modes include 

a first mode wherein said means for recognizing speech 
input recognizes speech as text, 

a second mode wherein said means for recognizing 
speech input recognizes speech as a command, 

a third mode wherein said means for recognizing 
handwritten or hand -drawn input recognizes hand- 
written input as text, 

a fourth mode wherein said means for recognizing 
handwritten or hand-drawn input recognizes hand- 
drawn input as a graphic, and 

a fifth mode wherein said means for recognizing hand- 
written or hand-drawn input recognizes handwritten 
or hand-drawn input as a command. 
40. An electronic whiteboard for processing handwritten 
input or hand-drawn input and speech input comprising: 
means for recognizing handwritten or hand-drawn input; 
means for recognizing speech input; and 
a mechanical switch for switching among operating 

modes, wherein said operating modes include 

a first mode wherein said means for recognizing speech 
input recognizes speech as text, 

a second mode wherein said means for recognizing 
speech input recognizes speech as a command, 

a third mode wherein said means for recognizing 
handwritten or hand -drawn input recognizes hand- 
written input as text, 

a fourth mode wherein said means for recognizing 
handwritten or hand-drawn input recognizes hand- 
drawn input as a graphic, and 

a fifth mode wherein said means for recognizing hand- 
written or hand-drawn input recognizes handwritten 
or hand-drawn input as a command. 
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[57] ABSTRACT 

A compact size, portable audio/display electronic apparatus 
having a compact mass memory with the capability of entry 
of user originated inquirable and inquisitorial input relative 
to currently displayed information and the random access 
capability relative to the compact mass memory in retrieving 
data responsive to such input. The apparatus includes a 
recording and reproducing device for recording and repro- 
ducing data in the form of alphanumeric information, dia- 
gram graphics, animation graphics, music, voice and other 
audio reproducible material. The reproducing device is 
employed only for the reproduction of the data, retrieving 
means for only retrieving the data for reproduction on a 
random access basis. Further, the display and reproducing 
device is arranged in one compact housing for the synchro- 
nized reproduction of at least two kinds of such data, e.g., 
the display of information and the audio reproduction of 
information directly associated with the displayed informa- 
tion. 

3 Claims, 7 Drawing Sheets 
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COMPACT PORTABLE AUDIO/DISPLAY 
ELECTRONIC APPARATUS WITH 
INTERACTIVE INQUIRABLE AND 
INQUISITORIAL INTERFACING 

This is a continuation of U.S. patent application Ser No. 
07/890,350, filed May 22, 1992 which is a continuation of 
U.S. patent application Ser. No. 07/496,788 filed Mar. 21, 
1990, now abandoned. 

BACKGROUND OF THE INVENTION 

This invention relates generally to a compact and portable 
audio/display device for displaying and reproducing data in 
form of atplanumeric information, diagrams, graphics, 
music, and sound which are recorded on a mass storage 
medium, such as, an optical medium, a magneto-optical 
recording medium, CD, CD-ROM or the like, wherein at 
least two kinds of data are randomly accessed for synchro- 
nized and uniform display and audio reproduction with the 
provision of a mass storage medium in a portable apparatus 
having an interactive user interface providing for inquirable 
and inquisitorial input by the user to random access and 
retrieve demanded response based upon information stored 
in such a memory. By "compact", it is meant that the 
apparatus is self contained, i.e., it contains all the functional 
components of the apparatus in a unitary handheld size unit. 
By "portable", it is meant that the apparatus can be com- 
fortably held by the user in one hand and operated by the 
user with the other hand, contains its own power source and 
is easily carried from one place to another, for dperation 
independent of any other kind of information source. 

Compact and portable type personal data audio/display 
apparatus are now in common use, for example, compact 
electronic pockctbooks, notebooks or organizers which 
include various functional combinations, for example, one 
or more of the following functions: note pad, diary, phone 
directory, calendar, world clock, as well as compact elec- 
tronic dictionaries and electronic translators. However, all of 
these types of compact and portable type personal data 
display apparatus are generally limited to alphanumeric and 
symbol data input/output. For example, a compact micro- 
computer notebook or pocket organizer is useful to collect 
dictated communications, such as, notes relative to an over- 
seas trip and the user may visually read written words or 
sentences that appear on the organizer's display, but for such 
a small slim device there is no facility to orally reproduce the 
voice pronunciation of the words and sentences due to its 
compactness. To include concurrent audio reproduction 
capability and user inquirable and inquisitorial interfacing 
while retaining their compact size is considered not practical 
under present technology for such organizers. 

Other compact and portable apparatus provide for con- 
current audio speech/display output. One such application is 
disclosed in Japanese Laid Open publication No. 62-279585 
published on Dec. 4, 1987. This publication discloses a 
portable CD and CD-ROM disc player apparatus which 
includes concurrent display of alphanumeric and symbol 
information with audio output, such as speech, associated 
with the displayed information. Both the audio output and 
visually displayed data is stored on the CD-ROM. Input 
means to the compact player apparatus provides interactive 
functions for stop, fast forward, fast return and start play- 
back command modes. However, this type of interactiveness 
is of limited capability, i.e., there is no provision of provid- 
ing a sufficiently large mass memory capability in combi- 
nation with an input means that permits a user to actively 
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interact with data concurrently displayed and audibly pre- 
sented to the user, i.e., permitting inquirable and inquisitorial 
interfacing with data stored in and display on the apparatus 
for the purpose of invoking a response thereto, such as is 

5 common practice with, for example, non-handheld personal 
computers (PC), workstations, etc. This is primarily due to 
the need for compact mass storage memory and the neces- 
sary electronics required to provide random access to the 
mass storage memory to locate, retrieve, decode and display 

10 and audibly present data in response to user inquiries. 

Another example of such a portable audio/visual elec- 
tronic apparatus is disclosed in U.S. Pat. No. 4,639,225. The 
portable apparatus utilizes an insertable ROM memory card 
which contains both audio and video information. The 

15 memory card is inserted into the device for display of 
information on the apparatus display in conjunction with 
reproduction of speech and other audio information, for 
example, such as the display and audio reproduction of 
novels, cartoons and dictionary terms. However, the ability 

20 of a portable video/audio electronic apparatus of this type to 
provide good graphic display quality or generate speech and 
other audio reproduction via an IC memory card circuit is 
generally of bad quality. Furthermore, there is no means 
provided by which a user may actively interact with the data 

25 presented on the insertable memory, such as inquirable and 
inquisitorial interfacing by the user relative to current dis- 
played data. Again, user interfacing in such compact and 
portable apparatus is aimed at a low level of controlling the 
presentation of the data, i.e., interfacing is limited to han- 

30 dling of the display and reproduction of audio information, 
such as start, stop, and sequential replay, either forward or 
reverse, of stored data, without the capability of entertaining 
inquiries by the user and providing information in the form 
of a response or answer to a presented inquiry. 

35 Thus, the disadvantage of employing an IC memory or IC 
memory card is that the memory capacity for graphics and 
audio reproduction is too limited and the sound quality 
reproduction is not acceptable. Moreover, the same is true 
with the use of a tape memory for sound which is undesir- 

40 able because random access is not efficiently possible and 
takes too long to retrieve and reproduce the selected sound 
portion. The use of a tape medium for recording takes a 
comparatively long period of time for random access of 
selected data or information desired for playback. Further, as 

45 previously indicated, a mass memory, such as a CD-ROM, 
may be easily employed as an external memory device for 
a PC, but these types of memories are inconvenient and 
expensive to consider for application apart from a personal 
computer application, such as for an application in a com- 

50 pact electronic organizer or other small portable electronic 
apparatus. An example of the foregoing is disclosed in U.S. 
Pat. No. 4,855,725 involving a portable electronic apparatus 
comprising an electronic simulated book that displays on a 
LCD panel a page of a book from a series of limited pages 

55 available in a limited memory provided in the portable 
apparatus. The apparatus provides for user input to display 
a selected page from a starting page to an ending page stored 
in the limited memory and also to consecutively display 
pages in ordered sequence set in memory. However, there is 

60 no ability to randomly select pages or input inquires into the 
apparatus based upon information displayed. Furthermore, 
since there is no mass storage capabilities in this portable 
apparatus, the user is required to be in close proximity for 
wireless communication to a PC which has peripheral mass 

65 storage of all the book pages on a CD-ROM. Thus, the 
limited supply of pages stored in the portable apparatus arc 
replenished from the remote PC. As a result, there is no 
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ability to employ the portable apparatus for long periods of cation to be inquisitory, to search and examine data or 

time at a remote location from the PC so that any true portions of audibly reproduced/displayed data and interact 

portability of the apparatus is limited by its proximity to a with the displayed data via iconic representations to obtain 

nonportable PC unit. In this sense, the display apparatus is additional displayed or audibly reproduced data, e.g., 

not truly portable 5 graphic animation, providing additional explanation and 

Furthermore, there is no means in the portable apparatus enhancement concerning displayed data, e.g., text, 

of U.S. Pat. No. 4,855,725 for the user to actively interact . ^ com P act electronic date display apparatus of this 

with the apparatus at a level of entering queries into the mve " u . on «™Pn«s recording and reproducing means for 

apparatus pertaining to concurrently displayed information ™ ordjn S ^ reproducing data m the form of alphanumeric 

«w > v Trr" 6 .. a, # j . lft information, diagram graphics, animation graphics, music, 

in orto to solicit audio responses and/or responsive dis- io ^ ^ ^ t udio Tepraducible materiaJ ^e^produci 

played dlustraUons to satisfy the entered query. Rather, as in . mcaDS fa j d J nX for mc rcproduction of L data, 

the case of previously discussed Japanese Laid Open pub- retri eving means for only retrieving the data for reproduc- 

lication No. 62-279585, there is conventional input means ^ on on a ^dom access basis. Further, the display and 

for the purpose of stopping, starting, a sequencing informa- reproducing means is arranged in one compact housing for 

tion displayed and audibly reproduced without further is the synchronized reproduction of at least two kinds of such 

higher level and more complex user interactiveness relative daU? e g f the of information and the audio repro- 

to the stored information. In this connection, while publi- duction of information directly associated with the displayed 

cation No. 62-279585 provides for a mass memory in the information 

form of a CD or a CD-ROM, the function of the CD-ROM fi ^ ng[n me data dis u and dala ^reducing me ans 

is to hold programs for their selective transfer to the memory 20 ^ ^ ^ ho {{ ^ ^ ^ ^ mvcntion to 

of the apparatus for execution by the apparatus CPU without miniaturize lhe entire a pp ara tus. Further, the employ- 

any capability of user inquirable and mquisitonal intcrac- mcnl rf aQ optica] mcdium rcJativc to ^ invcntion rcndcrs 

tiveness relative to data stored on the CD-ROM. it possibIe tQ display a large amolmt of data fa the fonn of 

Furthermore, it is difficult to input data to CD-ROM alphanumeric^ simple diagram and animation graphics and 

memories by means of an I/O device, such as, a compact 25 t 0 reproduce audio data, such as sound, voice and music, 

keyboard. Thus, a compact portable apparatus that provides bcing synchronized in presentation with the displayed data, 

complement keyboard input for inputting information are The present invention provides a simple, compact and 

small and not easUy useable due to their limited size for the portable electronic apparatus for recording visual and audio 

sake of compactness. Alternatively, a smaller key input data, randomly retrieving such data from the optical memory 

means to function as a user interface could provide desired 30 and synchronizing the displayed data with the audio repro- 

inquirable and inquisitorial interfacing with data stored in, duced data. Also, application software may be provided in 

displayed on and audibly reproduced by the apparatus. me optical memory or in the form of an IC memory in 

It is an object of this invention to provide a solution to the combination with the optical memory to perform various 

foregoing mentioned problems. ^ functions on demand. 

It is another object of this invention to provide inquirable Other objects and attainments together with a fuller 

and inquisitorial interfacing in a compact portable audio/ understanding of the invention will become apparent and 

display electronic apparatus. appreciated by referring to the following description and 

It is another object of this invention to provide a compact claims taken in conjunction with the accompanying draw- 

and portable audio/display electronic apparatus that pro- 40 ings. 

vides for a large mass memory capability and for a high level BRIEF DESCRIPTION OF THE DRAWINGS 

of interactiveness between the user and stored data in the _ „ . „ , 

apparatus capable of providing concurrent visual and audio FIG. 1 is an isome.r.c v.ew of an external configurahon 

data in response to inquiries provided by a user relative to fo / « he """P 30 ' portable audio/display electronic apparatus 

dis laved data invention including iconic input interfacing and 

* t . • j 45 associated display and audio output. 

It is another object of the present invention to provide a - . . , c ™„ - 

iJ . 1 J , j. . u u FIG. 2 is a cross sectional side elevation of FIG. 1 

compact display and reproducing apparatus which can , 1t „ ... , . 

j. . . . • L * *• 1 . ca . • c illustrating a first embodiment relative mass memory access, 

immediately retrieve a substantial amount of data or infor- B J 

mation recorded on small optical media by randomly access- FIG 3 15 a cross sectional side elevation of FIG. 1 

ing such data or information and, further, synchronizes the 50 aiustratin g a second embodiment relative mass memory 

reproduction of at least two kinds of such data, e.g., the access, 

display of information and the audio reproduction of infor- FIGS. 4, 5, 6, and 7 are respectively isometric views of 

mation directly associated with the displayed information. four olher external configurations for the compact portable 

audio/display electronic apparatus of this invention. 

SUMMARY OF THE INVENTION $$ n G S. 8A, 8B, 9A and 9B are examples of illustrating an 

According to this invention, there is provided in a com- application of the interactive interfacing provided by this 

pact size, portable audio/display electronic apparatus having invention relative to the synchronized displayed data and 

a compact mass memory with the capability of entry of user reproduced audio via random access of data stored relative 

originated inquirable and inquisitorial input relative to cur- 10 mass storage memory. 

rently displayed information and the random access capa- 60 FIG. 10 is an isometric view of a further embodiment for 

bility relative to the compact mass memory in retrieving data the compact portable audio/display electronic apparatus of 

responsive to such input. By "inquirable", it is meant the this invention including a combination display and overly- 

capability of permitting the user relative to a particular ing transparent input key panel. 

application to inquire into a subject matter provided in the FIG. 11 is a diagram of the control circuit employed 

compact mass memory or subject a subject recorded in the 65 relative to the several configurations of the compact portable 

mass memory to inquiry. By "inquisitorial", it is meant the audio/display electronic apparatus comprising this inven- 

capability of permitting a user relative to a particular appli- tion. 
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FIGS. 12 and 13 are examples illustrating an application several keys 3*. Fourteen such keys 3' are shown in FIG. 1. 

of the interactive interfacing provided by this invention Two keys, 3A and 3B, are for scrolling to a previous display 

relative to an electronic book. or to the next display of previously selected data relative to 

keys 3'. Keyboard 3 is employed to provide, for example, a 

DESCRIPTION OF THE PREFERRED s two step interactively inquirable interfacing by the user by 

EMBODIMENTS permitting the user to initially select a prerecorded subject 

n - , . - ••!•«« group prerecorded on medium 9, display the subject group 

Reference is now made to FIG. 1 which illustrates an * nd ^ rcaftcr xkci a parlicular it £ m J in ^ J * fo F r 

external view of one embodiment of a compact portable fortber lreatmcnt and display< Therefore, it is possible, for 

audio/display electronic apparatus 1 comprising this inven- example, with keyboard 3, to first choose a subject group by 

tion. Apparatus 1 includes casing 43 having display 2 on its 10 selecting an appropriate symbol marked key 3' followed by 

top surface and an input keyboard 3 for operating display 2. selecting a particular item in the displayed subject group for 

Display 2 is preferably a liquid crystal display because of its further detailed display by selecting another symbol marked 

low power requirement for the display of data. Any state- key 3'. Alternatively, a second level selection (i.e., an item 

of-thc-art liquid crystal display can be utilized, for example, selected from a retrieved subject group) may be a reassign - 

a monochrome or multi-colored or full colored display. 15 ment of function of the keys 3' on keyboard 3 to perform 

On the front side surface 1A of casing 13 are operating different tasks as compared to an assigned function of the 

buttons 4 for turning on apparatus 1 and for accessing the same ^ys at the first level of selection (i.e., selection of the 

drive for a mass memory medium 9, such as, a CD, particular subject group). 

CD-ROM, or other such optical memory medium and a FIG 8 illustrates an example of inquirable interfacing via 

volume control 5 for controlling the volume of the audio 20 an application of compact portable audio/display electronic 

output at speaker output 7. Side face IB of casing 13 1 employed as a pocket notebook employed, for 

includes a headphone output terminal 6, speaker output 7 sample in recording data or information relative to an 

a pdt t i a overseas trip. In FIG. 8A, three Japanese sentences at 22 

and CKl output terminal 8. have beeQ previously reC0 rded on medium 9 concerning the 

As shown in FIG. 2, cap portion 14 is rotatably mounted M tr j p arjc j are prompted to be visually displayed as well 

relative the top surface of casing 13 adjacent to LCD display audibly reproduced as speech via selection of a symbol or 

2. Cap portion 14 may be released by an operating button 4 numerically identified key 3* on keyboard 3, such as a 

and lifted to its open position to permit the insertion or particular key or key combination indicative of the search 

removal of optical medium 9 from within the interior of location or functioning as a pointer in memory for the 

casing 13, which medium is positioned on a spindle of 3Q location of the subject group for purposes of retrieving and 

spindle motor 11 . Thus, cap portion 14 provides for easy decoding of the selected digital data. One Japanese sentence, 

insertion and removal of medium 9 relative to casing 13. In j tern or sentence (1), is then chosen by the user for retrieval 

FIG. 3, another embodiment for insertion and removal of relative to the subject group of sentences (1), (2), and (3) by 

optical medium 9 is shown. In this case, cap portion 16 is selection of another symbol or numerically identified key 3' 

rotatably mounted relative to the bottom side 15 of casing 3S 0 n keyboard 3. Then, the chosen sentence (1) is translated by 

13, i.e., on the side opposite to display 2. Cap portion 16 may apparatus 1 into English, as indicated in FIG. 8B, and 

be lifted to its open position to permit the insertion or thereafter the English translated sentence (1) is displayed on 

removal of optical medium 9 from within the interior of display 2 as indicated at 23 along with audibly reproduced 

casing 13 positioned on a spindle of spindle motor 11. Thus, data in the form of speech vocalizing the English sentence 

cap portion 16 provides for easy insertion and removal of AQ w j t jj g00 d audio quality at speaker 7. 

medium 9 relative to casing 13 while being positioned in a FIG. 9 is another example of an inquirable interfacing 

region on casing 13 separate from the region of display 2 so wherein a directory has been provided in apparatus 1 on 

that the structure of display 2 is not stressed by pressure me dium 9 to include a list of hotels for a particular city, and 

applied relative to the opening and closure of cap portion 16. this subjcct group may be selected via a part icular key 3\ 

In FIGS. 2 and 3, schematically shown is a circuit board 45 sucn aSj key 3C in FIG. 1, designated as a pointer memory 

as supported in casing 13 from impact and shock and upon f or one 0 f a particular subject groups. Then, the list of hotels, 

which is the IC circuit for control circuit system 12, illus- illustrated at 24 in FIG. 9A, is displayed on display 2. At this 

trated in FIG. 11. System 12 controls the operation of point, the function assignment of keys 3' are changed 

medium 11 and the recording and retrieving of data relative relative to the selection of an item from the displayed subject 

to medium 11 via optical pickup 10 for display and audio 50 group. The user then selects a particular item, i.e., a hotel by 

reproduction. selecting a numerically identified key 3' on keyboard 3 

In connection with both embodiments in FIGS. 2 and 3, indicative of that selection, in this case, the hotel indicated 

the size of optical medium 9, e.g., a CD-ROM, is preferably as hotel (1) from the list 24. The graphic location of this 

small such as only 8 cm in diameter, which is sufficiently hotel is thereafter displayed as a simple map 25 on display 

small in size for compact portable apparatus 1 so as to be 55 2, which map is stored on medium 9 and is accompanied 

easily carried in one's hand. However, it is also possible to with reproduced audio explaining the location of the hotel, 

use a larger standard size CD, e.g., 12 cm, in order to support for example, on First Street relative to corresponding cross 

a larger display 2 thereby rendering it possible to display not streets in the city. Thus, optical medium 9 provides for a 

only alphanumerics but also to display larger area graphics, compact medium for storing a comparatively large amount 

such as a large map or the like. 60 of data or information which can be displayed on apparatus 

Input keyboard 3 may be a transparent touch panel or display 2 accompanied with necessary translation or graphic 

keyboard comprising individual keys 3' and also includes a diagrams concurrently reproduced with sound or music 

card inserlable card beneath the transparent top of keyboard whereas conventional compact and portable apparatus of 

3 for making designated changes to the nomenclature or this kind provide only for the display of alphanumeric type 

symbols employed in conjunction with the key caps of 65 data. 

keyboard 3. Keyboard 3 need not be a full complement FIGS. 12 and 13 illustrate an example of inquisitional 

alphanumeric keyboard and may consist, for example, of interfacing relative to apparatus 1 relative to an electronic 
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book comprising a series of electronic pages recorded on embodiment, input keyboard 18 is removably attached to the 

medium 9. Here, individual pages of text of the book, sucb forward underside of casing 13 in a recess 13 A. Keyboard 

as a page 61 A illustrated at 61 in FIG. 12, may be displayed 18 may be attached to the back side of recess 13 A by first 

on display 2 and for inquisitional embellishment of the text positioning keyboard 18 into recess 13A and then moving it 

for enlightenment of the user is provided by employing a s laterally as indicated by arrow 18A to bring about its 

transparent coordinate touch panel 26 in FIG. 10 to provide C0U pling to casing 13. Keyboard 18 may be connected as an 

a series of displayed icons, 63, 64 and 65, that are displayed mput to system 12 by means 0 f a cable (not shown) or by 

with the displayed text 61A and appear to the user through mcans of IR transceivers as is known in the art, e.g., 

the transparent touch panel 26/ The activation of these illustrated m tj. S . Pat. No. 4,855,725, previously discussed, 

displayed icons permits the user to receive additional rel- 10 ^ data storcd on mcdium 9 m apparatus x ^ displayed 

evant data related to currently displayed data, such as text oa display 2 may also be printed out ^ a ^ attached 

61A, to enhance the interest and inquisitiveness of the user t0 connecting cable 19. Also, apparatus 1 may be employed 

as well as provide supplemental data to the user to function „ an extemal memory device for yQ connection to other 

as a tutorial and enhance the knowledge and further under- office autom ation apparatus, e.g., personal computers, word 

standing of the user, with respect to the subject of the 15 processors or the like, by connecting such office apparatus to 

displayed text 61A. The icon bar indicator 63 shows the apparatus 1 via connecting cable 19. 

relationship of page 61 relative to entire electronic book of , . ... , f ^ j_ • • 

fjL *l i tL *l « tL 4 , L FIG. 6 is an embodiment of apparatus 1 wherein it is 

paces as represented by the length of bar 63 so that the user ... 4 , . , , rr A . , 

f* i , z ■ . . i . possible to display visual data concurrently with audio data 

has a visual concept of the number pages in the entire book. . ■ . * . . , . J _ A 

u , l j 7 o l f ! k A via receipt of a transmitted signal at antenna 20 to a 

The hatched portion 62 A of indicator bar 63 is a visual - n , .„ r . t 7L c 4 . 

. C K . , , j. , 20 transmitter/receiver m system 12 of apparatus 1. As an 

indication of the number the pages that proceed the display . . * , V L 

f ~ , r /! . r , r - . . r \ example, in a zoo, a user of apparatus 1 may approach a 

of page 61 and portion 62B of indicator bar 63 is a visual / ' . ' , . rr *u . • • 

. . c t . . c .u . c caged zoo animal and also come within the transmission 

indication of the number of pages that come after page 61. <• * „ A • i i ■ li . 

™ ... - r* • %~t . i . range of a transmitted signal at the animal cage receivable at 

The display of page 61 includes one stationary graphic ,° - A . j . . . 4 r , . , 

A, i . ha a €>• i u- • e antenna 20. The received signal provides both displayed and 

image 60 alone with text 61 A. Simple graphic animation of ~«. , . , t . t . . 5 , , ,, 

a l j* i 1 1 i j- i j - i 25 graphic data relating to the particular animal as well as an 

image 60 can be displayed by selecting displayed icon 65 to ° / . .. r . r . t , , ... 

4 • • • *■ u audio explanation that accompanies the displayed data as 

cause the stationary image 60 to move in animation such as „ *T, , , ^ . . , ,. . r . , 

, , r • *• r well as other data pertaining to the particular animal, 

through two or move motion frames, e.g., an animation of a . c 

sail boat moving over the waves on a body of water. Audio FIG * 7 IS an embodiment of apparatus 1 which enables 
speech may accompany the display of page 61 Sy selecting 30 ™°J??& and reproducing of data via the employment of a 
voice icon 64. For example, reproduced audio speech may ^ IC memorv card 21A msertable into slot 21B in front 
footnote or exemplify the displayed text. It is possible fo face 1A as weU as an optical medium 9, such as CD-ROM, 
accelerate the electronic aaVancemeht and display of pages intertable into apparatus 1 in the manner explained previ- 
by more than several pages at one time in conjunction with ousl y connection with the embodiments of FIGS. 2 and 3. 
bar 63 by the user finger pressing on transparent coordinate 35 ^ i{ » P ossiblc to rccord ncw data 00 mcdium 9 ' 
positions along the length of iconic indicator bar 63 to cause employing either an erasable type magneto-optic medium or 
selection and retrieval of a different book page either as a a wntc oncc lv P c medium, supplied from an external data- 
previous page within the confines of bar portion 62A or base v,a > for example, a modem. 

future page within the confines of bar portion 62B. FIG. 10 shows another embodiment of the electronic 

In FIG. 13, data is written into mass memory medium 9, 40 apparatus wherein LCD display 2 comprises an overlying, 

such as, a magneto-optical memory of apparatus 1 and transparent touch key panel 26. When a portion of the 

pointers 66 may be recorded in conjunction with recorded coordinate detection surface representing keys 3* is 

data, e.g., text 61A, that is considered particularly important depressed, as indicated in FIG. 10, the coordinate value of 

in nature relative to a particular data section, for example. that portion is delected and is representative of the particu- 

Po inters 66 are also retained in memory and corresponding 45 larly selected key 3'. 

encoded pointers to this data in memory is employed for FIG. 11 is a circuit diagram of the control circuit system 

retrieving at a later time and displayed for viewing, as is 12 employed with respect to the embodiments of apparatus 

known in the art. Such marked data may also be printed via 1 shown in FIGS. 1, 4, 5, 6, 7 and 10 and relative to the 

a connected printer as explained in connection with the previously explained interactive inquirabte and inquisitional 

embodiment of FIG. 5, discussed later. Another application 50 interfacing discussed in connection with FIGS. 8, 9, 12 and 

of pointer marked data is to provide the function of selec- 13. 

lively enlarging displayed alphanumeric information which [ n piG. 11, compact mass storage record and playback 

enables users with poor eye sight to more easily read such uri j, 27 is connected in system 12 to digital signal processor 

information. 30 through READ amplifier 28 and WRITE amplifier 29 and 

FIGS. 4 through 7 disclose other types of exterior con- 55 motor/actuator driver 50 is connected to optical pickup 

figurations that may be employed relative to compact por- actuator 10A and spindle motor 11 in unit 27. The heart of 

table audio/display electronic apparatus 1. In FIG. 4, display circuit system 12 is system controller 49 followed by digital 

2 is formed as part of a cap portion 14A over casing 13 and signal processor 30. System controller 49 contains the CPU 

cap portion 14A is rotatably attached at 14B to the upper and controller functions for executing instructions from 

portion of casing 13. Further, optical medium 9 is inserted 60 system memory 68, which includes both RAM and ROM 

in a slot 17 provided in the front side face IA of casing 13. memory. Functional operation of controller 49 is well known 

In FIG. 5, display 2 is arranged on casing 13 separately in the art, and processing input signals in accordance with a 
from a cap portion 14C and cap portion 14C is rotatably program stored in the ROM to control other processing 
attached to the upper portion of casing 13. Optical medium circuits to be noted later and controlling the access, time 
9 is inserted into apparatus 1 by opening cap portion 14C, as 65 sharing and synchronization of other circuit and I/O corn- 
indicated by arrow 14D, and placing medium 9 on the ponents connected in system 12 to data bus 70. For the 
spindle support positioned beneath cap portion 14C. In this purpose of simplification, address buses and addressing 
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functions necessary to the operation of system controller 49 entry is read by. system controller 49 in communication with 

and addressed components connected to bus 70 are not input interface via bus 70. 

shown but are well known in the art Internal bus interface System 12 may also include, for certain additional pro- 

54 controls access of connected components to bus 70 under gram applications, a loading system 48 connected to system 

the control of system controller 49. DMA provides for direct s controller 49 and to disc loading unit 47 for operation and 

peripheral to memory communication and control as is control of a microdisc, e.g., a small sized CD or CD-ROM, 

known in the art. inserted into unit 47. 

Connected to bus 70 is servo processor 51 which is Digital data processing unit 67 is connected to bus 70 for 

connected to servo driver 50 for controlling the movement ECC or interleaving processing. 

and operation of spindle motor 11 and pickup actuator 10A. 10 Lastly, power unit 52 provides for power connection to a 

Also, digital signal processor 30 is connected to bus 70 and DC power source such as a standard or rechargeable battery 

to servo signal processor 51 for the purpose of accessing or an automobile battery via a DC/DC adaptor or to an 

recorded data or recording data on medium 9 under the external AC source via an AC adapter, 

control of system controller 49. As indicated above, various different applications may be 

The function of digital signal processor unit 30 is inter- 15 employed relative to the compact portable audio/display 

leaving processing and to code data to be recorded on electro nics apparatus 1 of this invention as all exemplified in 

medium 9 via WRITE amplifier 29 and decode data FIG. 11 but provided in different possible combinations as 

retrieved from medium 9 via READ amplifier 28. READ indicated relative to the different configurations of FIGS, 

amplifier is all that is required relative to the employment of 1-7 and 10. Also, by providing a permanently installed 

mass memory medium 9 as a CD-ROM. Video data com- 20 read/write optical medium 9 in unit 27 wherein data can be 

pression is known in the art and is exemplified in U.S. Pat. provided to the memory from an external source, such as 

Nos. 4,868,764 and 4,868,653. Digital signal processor 30 input devices 3, 26, 58 and 59 or from input via IC card 40 

retrieves audio data, such as, voice or music, as well as or via antenna 46. Lastly, the configuration of apparatus 1 

display data and separates the data as coded and decodes, can be constructed to be thin in size thereby reducing its cost 

demodulates and decompresses the separated data. Audio 25 and improving its structural resistance against shock and 

data pass through digital filter 31, D/A converter 32, audio vibration. 

signal processing unit 33 and audio amplifier 34 for output, While the invention has been described in conjunction 

such as speaker output 7, or to a headphone set, via output with several specific embodiments, it is evident to those 

terminal 6 or to other remote audio output. skilled in the art that many further alternatives, modifica- 

Decompressed display data, such as alphanumeric or tions and variations will be apparent in light of the forgoing 

animation graphic data is provided on bus 70 to display description. Thus, the invention described herein is intended 

processor 36 via display memory 35 for processing and to embrace at such alternatives, modifications, applications 

display of the image data on LCD display 2. The display data and variations as fall within the spirit and scope of the 

may also be provided at video output for supply to an appended claims, 

external TV. What is claimed is: 

The function of apparatus 1 can be expanded by the l ; ^ interactive, audio-visual electronic apparatus corn- 
provision of additional memory employing a RAM IC card pnsing: 

40 (21A FIG. 7) to include a software program or data for a compact portable casing held in one hand by a user and 

operation relative to system controller 49. Such programs 40 interactively operated with the other hand of a user, 

are transferred via extended interface 39 to RAM system a liquid crystal display supported on a surface of said 

memory 68 wherein the program can be executed under the casing having a viewing surface for display of groups 

control of system controller 49. Also, the content on display of matrix arranged visually displayed elements as well 

2 can be provided for printout to printer 41 via extended as pictorial display of information, 

interface 39. 45 digital processing means in said casing for writing and 

Data may be received or transmitted through a modem reading audio-visual data to and from a memory in 

from an external data source employing external communi- response to user invoked input via a touch key input for 

cation interface 42 and communication cable connector 43. pictorial display of said information concurrently with 

Circuits 44 and 45 are relevant to the configuration shown in reproduced audio, 

FIG. 6 wherein data is transmitted to antenna 20 and 50 said touch key input comprising a transparent matrix 

processed by high frequency processing circuit 45 for stor- formed over said viewing surface of said liquid crystal 

age on medium 9 via digital signal processor 30 or for display, said matrix comprising a plurality of matrix 

display via display processor 36 and audio output via arranged transparent touch keys in corresponding 

processor 30. As an example, medium 9 may contain a alignment with said matrix arranged visually displayed 

portion of a map for display on LCD display 2 while a signal 5s elements of said liquid crystal display, 

received at antenna 20 may be superimposed on display 2 to each of said touch keys functionally operating as a first set 

pinpoint a particular location on the displayed map. By the of input keys to said digital processing means in 

same token, data on medium 9 may be read and decoded via conjunction with a first set of said matrix arranged 

digital signal processor 30 for transmission wherein the visually displayed elements and each of said touch keys 

prepared data signal on bus 70 is modulated at circuit 44 and 60 functionally operating as a second, different set of input 

is transmitted from antenna 20 by high frequency signal keys to said digital processing means in conjunction 

processing circuit 45. As an example, such a transmitted with a second, different set of said matrix arranged 

signal can indicate a particular location of apparatus 1 visually displayed elements after selection of at least 

relative to a remote transceiver one of said touch keys functionally operating as said 

User interactive input to system 12 is by one or more input 65 first set of input keys, 

devices, such as touch panel 26 (FIG. 10) or keyboard 3 or a compact mass memory contained within said casing 

18 (FIGS. 1-7), or mouse 58 or joy stick 59, wherein data comprising an optic storage medium for recording, 
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storage and retrieval of audio/display digital data com- 
prising a plurality of first data groups having a plurality 
of second data groups each of which is related to at 
least one of the first data groups, said second data 
groups comprising audio/visual data representations in 
response to an inquirable or inquisitorial query by a 
user through said transparent touch key input matrix 
relating to the plurality of first data groups, said inquir- 
able query permitting a user relative to a particular 
application to inquire into a subject matter provided in 
said compact mass memory, said inquisitorial query 
permitting a user to search and examine data and 
interact with the displayed data via iconic representa- 
tions to obtain additional displayed or audibly repro- 
duced data, 15 
said first set of said matrix arranged visually displayed 
elements being provided for selecting one of said 
plurality of first data groups via said digital processing 
means and said second set of said matrix arranged 
visually displayed elements being provided for select- 20 
ing one of said plurality of second data groups via said 
digital processing means relating to the plurality of first 
data groups; 

whereby when said touch keys are functionally operating 
as said second set of input keys, the second set of input 
keys operate in conjunction with said second set of 
matrix arranged visually displayed elements for allow- 
ing a user to select one of a plurality of said second data 
groups which comprise audio-visual representations. 

2. An interactive, audio-visual electronic apparatus com- 
prising: 

a unitary, handheld casing having at least one substan- 
• tially planar surface, 

internally housed, removable optical storage means for 35 
storing a plurality of data groups in a random access 
format, each of said data groups comprising interre- 
lated text, animation, audio and keymenu data, 

a keypad for capturing user input, said keypad including 
a plurality of multifunctional, data specific keys pro- 40 
grammable according to said keymenu data, 

visual display means disposed on said enclosure surface 
for displaying at least one screen of text and animation 
data of a first selected data group retrieved from said 
optical storage means, 45 

internally housed audio reproduction means for audibly 
reproducing audio data of said first selected data group 
synchronized to said at least one displayed screen of 
said first selected data group, and 

50 

an internally housed central system controller electroni- 
cally coupled to said keypad, optical storage, visual 
display, and audio reproduction means for program- 



25 
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ming said data specific keys of said keypad in response 
to user input of a selection from said keymenu data of 
said first selected data group, retrieving a second 
selected data group, including interrelated text, 
animation, audio and keymenu data, in response to user 
input selection, and outputting said second selected 
data group data to said keypad, visual display and audio 
reproduction means, 
wherein each of said plurality of data specific keys are on 
a touch sensitive matrix arranged transparent key pad 
and functionally operate as a first set of input keys in 
conjunction with said first selected data group and each 
of said plurality of data specific keys functionally 
operate as a second set of input keys in conjunction 
with said second selected data group. 
3. A portable multimedia device, comprising: 
a handheld casing housing an information processor and 
a mass storage unit in communication with said infor- 
mation processor, said mass storage unit including a 
medium storing multimedia data contextually arranged 
in a hierarchy defining a plurality of subject groups, the 
multimedia data comprising at least one of visual and 
audio data; 

a display disposed on said casing and communicating 
with said information processor for displaying visual 
information to a user, the visual information including 
a plurality of interfacing icons; 

an audio reproduction unit communicating with said 
information processor for reproducing audio informa- 
tion to the user; and 

a transparent touch panel disposed on said display, said 
touch panel defining touch keys aligned with the inter- 
facing icons, user activation of at least one of the touch 
keys causing said information processor to select one of 
the subject groups of the multimedia data hierarchy; 
and 

subsequent user activation of at least one of the touch keys 
permits a user to inquirably and inquisitorily access 
multimedia data stored on the medium of said mass 
storage unit based on the selected one of the subject 
groups of the multimedia hierarchy and at least one of 
the visual information being displayed on said display 
and the audio information being reproduced by said 
audio reproduction unit, said inquirable query permit- 
ting a user relative to a particular application to inquire 
into a subject matter provided in said compact mass 
memory, said inquisitorial query permitting a user to 
search and examine data and interact with the displayed 
data via iconic representations to obtain additional 
displayed or audibly reproduced data. 
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